Make the two build scripts importable, and read the real prefix lists - #108
Merged
Merged
Conversation
This was referenced Sep 22, 2026
realmarcin
force-pushed
the
fix/importable-pipeline-scripts
branch
from
September 22, 2026 07:26
0955705 to
0bbe30b
Compare
`build_subsets.py` resolved every Mech checkout at import — `MECHS` called `mech_root`, which exits when one is missing — and then scanned. `build_data.py` read both JSON inputs at import and rewrote fleet_data.json. Neither could be imported to look at a constant (#97). Both now keep only the pure constants at module level. `build_subsets` grew a `prepare()` that resolves the checkouts and indexes HabitatMech's pages, and everything from the scan onward moved into `main()`; the function bodies are untouched and still refer to the same module-level names, so `MECHS`, `hab_pages` and the rest are declared empty and filled by `prepare()` rather than moved. `build_data` grew `build()` and `main()`, with the same treatment. Importing either is now free and writes nothing. That lets the tests read the lists the pipeline actually uses. They had been recovered from the source text with `ast`, which cannot see a list rebuilt after its literal — the hole #99 closed partly and #101 recorded the rest of. Reading the objects retires the class: all five shapes now fail, including the three the parser could not see. VOC, _X = [...], 1 parser: passed import: FAILED del VOC[0] parser: passed import: FAILED VOC[0] += "X" parser: passed import: FAILED VOC += [...] parser: FAILED import: FAILED VOC.append(...) parser: FAILED import: FAILED `literal()` and its mutator allowlist are gone with the parsing they served. Verified behaviour-preserving by running both scripts before and after the refactor over the same checkouts and comparing every output file. `assets/fleet` and `subsets_summary.json` match semantically: identical per-Mech scan counts, identical edge weights and `by` breakdowns, identical term lists in identical order, every term entry equal. One edge file differs by bytes, and it is not this refactor. `CAS` and `ENVO` are tied at 1 in that edge's `by`, and `Counter.most_common()` breaks ties by insertion order, which comes from iterating a set of strings under per-process hash randomization. Filed as #107, since it means a refresh churns committed assets for no reason and the pipeline cannot be used to verify itself. No committed data changes here; the runs' output was restored. Rebased over #105, carrying its CITATION filter into main(). From the review: main() now calls prepare() itself, so importing the module and calling main() resolves the checkouts instead of failing with a KeyError inside scan() (closes #117); the unused `import ast` left behind by literal()'s removal is gone (closes #116). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
realmarcin
force-pushed
the
fix/importable-pipeline-scripts
branch
from
September 22, 2026 07:26
0bbe30b to
0c7875f
Compare
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #97 and #101.
The problem
build_subsets.pyresolved every Mech checkout at import —MECHScallsmech_root, which exits when one is missing — then scanned the corpora.build_data.pyread both JSON inputs at import and rewrotefleet_data.json. Neither could be imported to look at a constant.The refactor
Both keep only pure constants at module level.
build_subsetsgrew aprepare()that resolves checkouts and indexes HabitatMech's pages; everything from the scan onward moved intomain(). Function bodies are untouched and still refer to the same module-level names, soMECHS,hab_pagesand friends are declared empty and filled byprepare()rather than moved into it.build_datagrewbuild()andmain().Importing either is now free and writes nothing.
The payoff: #101 retired, not narrowed
The tests had to recover
VOCandPREFfrom the source text withast, which cannot see a list rebuilt after its literal. That was the hole #99 closed partly and #101 recorded the rest of. Reading the real objects retires the class:VOC, _X = [...], 1del VOC[0]VOC[0] += "X"VOC += [...]VOC.append(...)literal()and its mutator allowlist are deleted along with the parsing they served.Verified behaviour-preserving
Ran both scripts before and after the refactor over the same checkouts and compared every output file.
assets/fleetandsubsets_summary.jsonmatch semantically: identical per-Mech scan counts, identical edge weights andbybreakdowns, identical term lists in identical order, every term entry equal.One edge file differs by bytes only, and it is not this refactor:
Same length, same content.
CASandENVOare tied, andCounter.most_common()breaks ties by insertion order, which comes from iterating a set of strings under per-process hash randomization. Filed as #107 — it means a refresh churns committed assets for no reason, and the pipeline cannot be used to verify itself.No committed data changes here; the verification runs' output was restored.
27 tests and
assemble_page.py --checkpass.Review fixes
Rebased over #105, carrying its
CITATIONfilter intomain(). From the adversarial review:main()now callsprepare()itself, so importing the module and callingmain()resolves the checkouts instead of failing with aKeyErrorinsidescan()(closes #117); the unusedimport astis gone (closes #116). Left filed: #118 (unclosed handles and non-atomic writes).🤖 Generated with Claude Code